1. Field of the Invention
The present invention relates to an apparatus and method for aligning images by detecting features. More particularly, the present invention relates to an image processing apparatus applied in an imaging device such as a digital camera, etc. which electrically shoots an object using an image pickup device such as a CCD and CMOS, etc.
2. Description of the Related Art
When shooting a dark object such as a night view, etc. using an imaging device such as a digital camera etc., in addition to opening an aperture and delaying a shutter speed, it is effective to enhance photographing sensitivity, namely, to enhance signal amplification of an image pickup device such as a CCD and CMOS, etc. However, if the photographing sensitivity gets too high, a noise such as an internal noise of the image pickup device will increase and an image becomes visually undesirable.
Accordingly, Japanese Laid-Open (Kokai) Patent Application No. 2004-357040 discloses an art (hereinafter referred to as conventional prior art) for reducing the noise by synthesizing a plurality of images that are taken in series. This is based on the findings that when a plurality of images that are taken of a same object in series are synthesized by overlapping them, random noise components are averaged while the object section of the synthesized image increases in proportion to the number of images. This enables reducing noise and enhancing photographing sensitivity, thereby enabling improvement of shooting performance of a dark object.
Meanwhile, due to a generally slow shutter speed when taking an image of a dark object, camera shake may occur. In addition, in the case where images of a same object are taken in series, a subtle shift may occur in the camera composition at each shoot timing.
Accordingly, merely “synthesizing a plurality of images that are taken of a same object in series by overlapping them” is not sufficient and it is necessary to enhance overlapping accuracy, namely, to perform precise alignment of the subject images. Regarding this aspect, the conventional prior art specifies that it extracts a “feature point” from a plurality of images that are taken of a same object in series and performs alignment (i.e., coordinate transformation) such that the feature points from these images are overlapped with each other.
In the conventional prior art, a “feature point” is defined as a “point at which luminance is high.” However, it refers more generally to a point for which a motion vector (i.e., an optical flow) can be determined because a pixel value changes significantly when the pixel is moved along any line.
However, the conventional prior art as described above has a problem which should be improved.
FIG. 18 is a diagram showing an example of an object for explaining inconvenience caused by the conventional prior art. This object example comprises a sky 1 as a background, a building 3 having a number of windows 2 on the left, a utility pole 4 on the right, and electric cables 5, 6 that are stretched from and across the pole 4 from the top to the right bottom of the diagram. A dotted frame 7 on the left side of the diagram shows a photograph composition which does not cause inconvenience in synthesizing images (hereinafter referred to as a first composition 7) and a dotted frame 8 in the upper side of the diagram shows a photograph composition which causes in-convenience (hereinafter referred to as a second composition 8).
FIG. 19A is a diagram showing feature points in the first composition 7, and FIG. 19B is a diagram showing feature points in the second composition 8. The conventional prior art as described above extracts points with a high luminance in the image as the feature point. Accordingly, points with high luminance, that is, a corner of the outer wall of the building 3, a corner of the window 2 and the like are extracted as feature points both in the first composition 7 and the second composition 8 (See black circles in the diagram).
The difference between the first composition 7 and the second composition 8 lies in unevenness in distribution of the feature points. In other words, they differ in that while the feature points are uniformly distributed in the image in the first composition 7, the feature points are distributed unevenly in a part of the image (i.e., at the left bottom corner in this diagram) in the second composition 8.
In this example, when a plurality of images that are taken of a same object in series are synthesized, alignment has to be performed only at one section of the image (such as a center of the image, etc.) in principle. In a practical point of view, however, because there are a multitude of factors that may contribute to a minute error in alignment such as image distortion caused by peripheral aberration of the photographing lens, the limit of the image resolution, fluctuation of the light source, deformation of the object itself and the like (hereinafter referred to as an alignment error factors), alignment is essential at various points in the image.
In light of the foregoing points, when attention is paid to the first composition 73 and the second composition 8 as described above, since the feature points exist evenly in the image in the first composition 7, alignment can be performed without any inconvenience when a plurality of images are synthesized. On the other hand, since the feature points exist only in a part of the image in the second composition 8, precise alignment is possible only in the corresponding position (i.e., the portion where the feature points exist) when a plurality of images are synthesized. Accordingly, for the remaining portion (i.e., the portion where the sky 1 and the wire cables 5, 6 appear), there is no choice but to perform alignment indirectly referring to the information on remote feature points. Accordingly, this is problematic in that the alignment error factors as described above may lead to misalignment in the remaining portion (i.e., the portion where the sky 1 and the wire cables 5, 6 appear).
Therefore, the first object of the present invention is to provide an image processing apparatus capable of extracting feature points evenly from all portions of the image.
The second object of the present invention is to provide an image processing apparatus capable of accurately tracking an object in all portions of the image.
The third object of the present invention is to provide an image processing apparatus capable of precise alignment at all portions of the image when images are synthesized.